{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Lab 7 - Samples and boxplots\n",
"\n",
"A *population* is everyone or everything that we want to study. For example, if we want to study the blood pressure of New Yorkers, the population would be everyone living in New York.\n",
"\n",
"A *sample* is the subset of the population for which we have observations/measurements/data. For example, for the blood pressure study, it's impractical to measure the blood pressure of everyone living in New York, so we might just randomly select 100 New Yorkers and measure their blood pressure. Those 100 people are the sample.\n",
"\n",
"In this lab, we will investigate how the distribution of a sample compares to the distribution of its population. Recall that the distribution of a sample or population is the precise description of the possible data values along with their frequencies (how often a value occurs). In previous labs, we learned how to visualize distributions using bar charts and histograms, and today we will learn a third way, called a *boxplot*.\n",
"\n",
"We will use the Green Taxi Trip dataset from Labs 3, 4, and 6 and the MoMA artwork dataset from Lab 5. \n",
"\n",
"As usual, we will import the matplotlib and pandas packages, and set plots to appear in the Jupyter notebook."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"import pandas as pd\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Samples of a quantitative variable (green taxi trip distances)\n",
"\n",
"We will use the green taxi dataset first, so read that CSV file into a dataframe named `taxi`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check that the dataframe was created properly by displaying it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's assume that this dataset is our *population*, meaning we are interested only in the taxi trips contained in this dataset and no other trips. The *distribution* of the trip distance for this population can be visualized by the histogram of the trip distance. Write code below to generate a histogram with 40 bins of the trip distances in `taxi`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Pattern:
\n",
" dataframe_name[\"column_name\"].hist()\n",
" \n",
"\n",
"Now we will take a random sample of 10 trips and plot the histogram of this sample. Type the code `sample10 = taxi.sample(10)` below and run it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This code randomly selects 10 rows from the dataframe `taxi` and copies them into a new dataframe called `sample10`. To see this, display the dataframe `sample10` below."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Make a histogram of the trip distances in the `sample10` dataframe below:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Answer:
\n",
" sample10[\"trip_distance\"].hist()\n",
" \n",
"\n",
"How does this histogram of the sample trip distances compare to the histogram of the population distances? \n",
"\n",
"Let's try a bigger sample of size 50. Write code below to take a sample of 50 rows from the dataframe `taxi` and plot the histogram of these sample trip distances."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Answer:
\n",
" sample50 = taxi.sample(50)\n",
"sample50[\"trip_distance\"].hist()\n",
" \n",
"\n",
"How does this histogram of the sample trip distances compare to the histogram of the population trip distances?\n",
"\n",
"What happens if you re-run this code? Why?\n",
"\n",
"Let's try an even larger sample size of 200. Write code below to take a sample of size 200 from the `taxi` dataframe and plot the histogram of these sample trip distances. Try different numbers of bins, and use the one that seems to give the most informative histogram."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"How does this histogram compare to the histogram of the population?\n",
"\n",
"Take a final sample of 1000 trips. Can you predict how the trip distance histogram will change from the 200 sample one? \n",
"\n",
"Plot the histogram of these 1000 trip distances below. Use the number of bins that gives the most informative histogram."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"How does this histogram compare to the histogram of the population? As the sample size increased, how did the histograms change?\n",
"\n",
"### Samples of a qualitative variable (gender of MoMA artists)\n",
"\n",
"Now we will look at samples from the MoMA artworks dataset from Lab 5. Load the CSV file into the dataframe `art`, and check that it was read in correctly."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We are going to look at the distribution of the artists' gender. First write code to count the number of each value in the gender column and display the counts. (Don't make a bar chart yet!)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Pattern:
\n",
" dataframe_name[\"column_name\"].value_counts()\n",
" \n",
"\n",
"What do you notice about the values? \n",
"\n",
"Some of the art works have multiple artists, so there are a lot of values beyond male and female. The code below will create a new dataframe called `solo_art` that contains only works by a single artist. We will learn how to write these filters in the next two classes, but for now, just run the code."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"male_filter = art[\"Gender\"] == \"(Male)\"\n",
"female_filter = art[\"Gender\"] == \"(Female)\"\n",
"neither_filter = art[\"Gender\"] == \"()\"\n",
"solo_art = art[male_filter | female_filter | neither_filter]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Write code below to count the number of artists with each gender in the new dataframe `solo_art`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Answer:
\n",
" solo_art[\"Gender\"].value_counts()\n",
" \n",
"\n",
"Next, make a bar chart of the gender counts (you will have to save the counts as a variable)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Pattern:
\n",
" counts_variable = dataframe_name[\"column_name\"].value_counts()\n",
"counts_variable.plot(kind = \"bar\")\n",
" \n",
"\n",
"What do you notice about this gender distribution? We use all MoMA artworks by solo artists as our population, and so this bar chart visualizes the population distribution.\n",
"\n",
"Now let's take a sample of 10 rows from the `solo_art` dataframe. Can you figure out how to do this?"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Answer:
\n",
" art_sample10 = solo_art.sample(10)\n",
" \n",
"\n",
"Write code to create a bar chart of the gender of the artists in this sample:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Answer:
\n",
" counts10 = art_sample10[\"Gender\"].value_counts()\n",
"counts10.plot(kind = \"bar\")\n",
" \n",
"\n",
"How does the distribution of this sample compare to the population distribution?\n",
"\n",
"Let's take a larger sample of size 50 from the `solo_art` dataframe and plot a bar chart of the genders in this sample."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Answer:
\n",
" art_sample50 = solo_art.sample(50)\n",
"counts50 = art_sample50[\"Gender\"]\n",
"counts50.value_counts().plot(kind = \"bar\")\n",
" \n",
"\n",
"How does the distribution of this sample compare to the population distribution and the previous sample?\n",
"\n",
"What happens if you re-run this code? Why?\n",
"\n",
"Now let's try a sample of size 200. Take the sample from the `solo_art` dataframe and make a bar chart of the genders in this sample."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"How does the distribution of this sample compare to the population distribution and the previous samples?\n",
"\n",
"Finally, let's try a sample of size 1000. Take the sample from the `solo_art` dataframe and make a bar chart of the genders in this sample."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"How does the distribution of this sample compare to the population distribution? How did the bar charts change as the sample size increased? Is this the same behavior you saw with the histogram of trip distances and the previous sample distributions?\n",
"\n",
"### Box plots\n",
"\n",
"Finally, let's look at another way to visualize the distribution of quantitative data with a *box plot*. Type the code `taxi[\"trip_distance\"].plot(kind = \"box\")` below and run it."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A box plot shows 5 aspects of a distribution:\n",
" - the minimum value\n",
" - the 25\\% percentile (25\\% of the data is less than this value)\n",
" - the median\n",
" - the 75\\% percentile (75\\% of the data is less than this value; equivalently 25\\% of the data is greater than this value)\n",
" - the maximum value\n",
" \n",
"Can you figure out what parts of the box plot correspond to these 5 numbers?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Challenges:\n",
"- What happens if you take an even larger sample size than 1000? (for either or both datasets)\n",
"- Rerun the code that takes a random sample and plot the distribution as a histogram or bar chart for sample size 50 and sample size 1000. How much do the plots change? Why?\n",
"- Make a box plot of another column with quantitative data."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
}
},
"nbformat": 4,
"nbformat_minor": 2
}